[FEATURE] Render every page to Markdown alongside HTML - #1410
Open
linawolf wants to merge 16 commits into
Open
Conversation
Tools that read our documentation programmatically want the content without the surrounding HTML. The reStructuredText source is not good enough: includes, substitutions and interlinks are unresolved there, and directives only become content during rendering. This is a render target on the same footing as HTML - same AST, same pipeline, its own templates. A new package typo3-docs-theme-md registers an "md" output format with 40 templates and its own node renderers, so RenderContext writes Feature.md beside Feature.html. The theme ships it, so no second extension has to be configured. Markdown is GitHub Flavored: admonitions and version changes become alerts, tables become pipe tables. CommonMark has no tables at all. It renders always. That cannot be done by prepending "output_format": an explicit value replaces the default ["html", "interlink"] rather than extending it, so a project configuring nothing would get Markdown and no HTML. Appending in the compiler pass leaves every other format untouched - a project rendering only "singlepage" or only "rst" keeps doing that. Naming "md" explicitly still wins, so a Markdown-only render stays possible. Links are permalinks built from interlink_shortcode, because a downloaded file has to keep working. A page linked without an anchor is in this same manual, so its document entry supplies one; "#" is the page itself. Anchors are normalised the way getPermalink() does it: a URL fragment keeps the casing of the element id, while the inventory registers the target reduced. Where no permalink can be built the link stays relative and that is logged. A toctree renders as the list of links it is, unless it is :hidden: - the same rule the HTML output applies. A node with no Markdown template leaves an HTML comment naming it, so the remaining gaps are greppable rather than silent. The HTML head points at the Markdown with rel="alternate", which supersedes #1401. Verified against the TYPO3 core changelog, 3912 entries: four confval directives left unrendered, no HTML entities, and all 12622 permalinks resolve to a target in the generated inventory. Rendering Markdown alongside HTML costs 8.7% (127s to 138s). Part of: #1400 Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019EuSAzF5qw81NzPusWy2hC
render-markdown="false" drops the Markdown files, the head link and the menu entries together. Adds a "Markdown" download button beside "Give feedback" and "Edit on GitHub", and "View as Markdown" in the options menu. The download is named from the manual and the page instead of the URL, where every overview page arrives as "index.md". The name comes from the document's label, not its title: titles repeat, so an entry backported to three versions shares one title anchor. Using it collided 29 of 3912 names and pointed a :doc: link at the wrong version. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019EuSAzF5qw81NzPusWy2hC
Source prose is not Markdown. Unescaped, "__dunder__" turned bold, "array[0](x)" turned into a link and a leading "#" into a heading. Only what can start a construct is escaped: an underscore inside a word is not emphasis in CommonMark, so escaping every one added 2287 backslashes to the core changelog for nothing. confval renders as a list item: indenting under a bare paragraph is an indented code block, not containment, so a definition had no visible end and nested confvals did not nest. The name is bold and the property labels italic; neither wraps a code span, which not every renderer parses. rubric becomes bold via a GeneralDirectiveNodeRenderer that looks up a template per directive name, the way HTML does. Text roles dispatch by name too, so ":code:" and the default role keep their markup. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019EuSAzF5qw81NzPusWy2hC
composer normalize sorts "T3Docs\Typo3DocsThemeMd\" before "T3Docs\Typo3DocsTheme\", and the Quality job checks it. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019EuSAzF5qw81NzPusWy2hC
…down Tabs, cards and accordions are boxes: panels behind a click, a bordered card, a collapsed stack. Markdown has none of that, so each title becomes bold and the contents follow one below the other. A card writes out its image, header, title, body and footer in that order. A footer given a block can hold a whole list, so it is rendered as blocks rather than collapsed onto one line. Three nodes were exempted from the fallback marker as structural, on the assumption that they pass their content through. They do not: rst-class dropped its whole content, and field and definition lists lost their terms. Each one now has a template. An image read its alt text and its path from properties that do not exist, so every image rendered as "![]()" pointing at an unnormalised path. It now reads the same options the HTML template reads. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A console command is a named definition with fields, like a confval, and is rendered like one: the signature bold as a list item, so the definition has an end and the arguments and options nest inside the command they belong to. Usage, arguments, options and help are italic labels, so the command outweighs the words introducing its parts. A command whose help text is empty gets no "Help" label. The overview becomes a pipe table. A command is linked only when it can be: the anchor of a noindex command is registered nowhere, so a permalink to it would lead to a 404 rather than to the command. An argument shows only its description, as in HTML. ArgumentNode also carries isRequired and a default, but CommandNodeService fills the first from "is_value_required", an option key, so it is always false, and gives the second the description. Rendering either would state something untrue about the command. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A ViewHelper writes out the parts its "display" option asks for, in that order, as in HTML: doc tags, prose, the link to the source and the arguments. A deprecated or internal tag becomes an alert, and so does the note about arbitrary arguments. An argument is shaped like a confval, which is what the HTML template builds it as too: the name bold, then type, default and required, then the description. "Required" reads as true rather than as the "1" that HTML prints for the boolean. "Arguments" is bold rather than a heading. The HTML template makes it an h2, but a section can hold several ViewHelpers, and a real heading would leave the outline claiming the arguments of the first one end where the second one starts. The Markdown parser emits a NullNode where there is nothing to render. Marking it claimed content was dropped where none exists, so it joins the nodes the fallback passes through untouched. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
These five directives are layout around content Markdown already has, or source text HTML turns into a picture. An hlist is a bullet list in columns and a glossary is its entries behind an A-Z bar; neither the columns nor the bar survive, the list and the terms do. A video cannot be embedded and the node carries nothing but the id, so it becomes a link to itself -- the watch URL, since the privacy gain of youtube-nocookie.com is about loading the player on our page, which a link does not do. A diagram and a formula are shown as their own source, fenced and tagged plantuml and math, which the renderers that know them pick up. YoutubeNode, UmlNode and MathNode are not GeneralDirectiveNodes, so the lookup by directive name does not reach them. Math is registered in the Markdown package and the other two in the theme, each where the package holding the node is already a dependency. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A directory tree is a nested list that HTML gives folder icons and collapse toggles. Markdown keeps the nesting and drops the rest. The items are not ordinary list nodes -- an item holds its label in its children and its sub-trees in "subLists" -- so the structure is walked the same way the HTML template walks it. Passing the node through ran every name in the tree together on one line. Each item emits exactly one newline. A blank line between them would make the list loose, and a renderer then wraps every name in its own paragraph. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
":file:`/etc/passwd`" rendered as bare text in Markdown while every other code-like role kept its backticks, so a directory tree showed its directories as code and its files as prose. A node renderer registered from a template matches with is_a(), so the first one that fits decides. FileInlineNode reaches InlineCompoundNode through AbstractLinkInlineNode, and the Markdown package maps that class generically. prependExtensionConfig() puts the last caller first, and the Markdown extension was prepended last, so its generic map came before the theme's own registration and swallowed the role -- leaving the node's children to render as plain text. The Markdown extension is now prepended first, which puts the theme's nodes ahead of the generic map. CodeInlineNode extends InlineNode directly and was never affected. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Classes, interfaces, traits, exceptions, enums and their members had no Markdown at all: the markers counted twenty, but the real loss was the members below them, which ran together into one line of prose -- "A class to do some returning.OtherLibrary\ReturnedClass..." Each definition is shaped like a confval, the way every named definition in this theme is: the signature bold as a list item, so the definition has an end and its members nest inside it. Short properties come before the description, blocks after it -- a fully qualified name is one line and leads, a method's "Returns" is prose and follows. A namespace renders as nothing, as in HTML, where it is an anchor and nothing else; the namespace is spelled out in every fully qualified name inside it anyway. The theme now requires t3docs/guides-php-domain, which it has been rendering all along through the dependency of another package. Four escaping bugs surfaced while building this. renderNode() is marked safe in Twig but a filter after it is not, so a link URL, a definition list term and a command table cell all reached the output as HTML entities. The rendertest held eighteen and now holds none. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A caption says which file a snippet belongs in, or whether it is the before or the after of a migration. Markdown dropped it, leaving consecutive blocks in the same language indistinguishable: the StringUtility::multibyteStringPad deprecation showed two bare php blocks where HTML labels them "Before (deprecated)" and "After (recommended)". It is now a bold line above the fence, where HTML puts it too. Found by comparing the rendered TYPO3 core changelog against its HTML: 620 code blocks in 273 of its 3912 files carry one, most of them a file path such as "EXT:my_extension/ext_localconf.php". Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…down Rendering the TYPO3 references, guides and tutorials turned up five directives the rendertest does not contain. Two carry content and were losing it. The manuals write a configuration value as "option", not as "confval", and a file definition as "typo3:file"; both ran together into a line of prose -- "stringEXT:my_ sitepackage/...The path to the Fluid template file". Both are now shaped like a confval, which is what the HTML templates build them as. OptionNode is not a GeneralDirectiveNode, so the lookup by directive name does not reach it; it is registered by class, as in HTML. The other three are controls, not content, and render as nothing or as their plain content: the "Edit on GitHub" button beside an included section, the main menu JSON that only the JSON output writes, and the list of languages the highlighter knows. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A permalink without a version resolves to the latest stable release. Every link in the Markdown of a 13.4 manual therefore led into whatever manual is current when the file is read -- the opposite of what the permalinks are here for, which is that a downloaded file keeps saying what it said. Manuals that exist only once take no version; DefaultInventories already knows which those are, and what it does not know is a third-party manual, which is versioned. A checkout names itself "main (development)", so the version is cut back to the bare "main" the URL wants. A version that is set but cannot go into a URL is reported once per document. A version that is absent is not: the theme treats it as optional everywhere else, and there is nothing the author could do about it here. Other interlinks are unaffected. They resolve through the inventory into a full URL that already names a version, taken from typo3-core-preferred. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A Markdown file says nothing about itself: which manual and version it came from, which source it was rendered from, when, and where its HTML lives. A tool that has the file has no way back to any of that. It now opens with YAML front matter carrying title, manual, version, permalink, source path and the render date, plus "start: true" on the manual's entry point and "noindex: true" for a page kept out of the search. Front matter is not CommonMark, but it is what every tool that reads Markdown expects, and this output exists to be read by tools. The permalink is the one built for every other link, version suffix and all, so it names this page independently of where the file ends up and leads back to the HTML it was rendered from. The source path comes from where the "Edit on GitHub" link takes it, so an included file names itself rather than the document that included it. The start page is recognised by comparing against the project's root entry, because "Index.rst" is a name several documents in a manual share. A key whose value is unknown is left out rather than written empty: a permalink with nothing behind it is worse than no permalink. Values are always quoted, or a title like "Feature: 12345" would parse as a mapping and a version like "13.4" as a number. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
linawolf
force-pushed
the
task/markdown-output-format
branch
from
September 13, 2026 09:33
0d06472 to
b256395
Compare
linawolf
marked this pull request as ready for review
September 13, 2026 10:06
The package was not listed anywhere the other five are, so it would never have been split into its own repository, and the theme required it as "@dev" while every other dependency inside this repository uses "self.version" -- a released typo3-docs-theme would have asked Packagist for a development version of a package that is not published there. The root composer.json now replaces it like its siblings, and the lock file carries the content hash that follows from it. Still to do outside this repository, per Documentation/Developer/ MonoRepository.rst: create TYPO3-Documentation/t3docs-typo3-docs-theme-md, give the php-based-rendering-bot team maintain rights, and register it on Packagist. The split has to run once on a push before a tag is made. Signed-off-by: lina.wolf Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1400.
Tools that read our documentation programmatically want the content without the
surrounding HTML. The reStructuredText source is not good enough: includes,
substitutions and interlinks are unresolved there, and directives only become
content during rendering.
A new package
typo3-docs-theme-mdregisters anmdoutput format on the sameAST and pipeline as HTML, so
RenderContextwritesFeature.mdbesideFeature.html. The theme ships it;render-markdown="false"switches it off.Markdown is GitHub Flavored — admonitions and version changes become alerts,
tables become pipe tables. Links are permalinks built from
interlink_shortcode, because a downloaded file has to keep working.Verified against 39 real manuals: the core changelog, the TCA, TypoScript,
ViewHelper and exception references, eleven guides and tutorials including
TYPO3 Explained, the 21 system extensions that ship their own documentation,
and the news and content-blocks extensions as third-party cases whose
interlink shortcode is a composer name. 10264 Markdown files.
comment naming it; the rendertest started at 243 of those, and every manual
now reports none.
exactly one page marked as its start.
version wherever the manual declares one: all 92 in the TCA reference and 382
of the 383 across the system extensions, none of the 3912 in the changelog,
which is published once.
about
htmlentities(),htmlparseror XML examples, where HTML shows thesame characters.
page by page, what is left is the HTML splitting identifiers across
syntax-highlighting spans, and controls: "Copied!", "Edit on GitHub", the
breadcrumb.
Rendering Markdown alongside HTML costs about 2%, which is less than the spread
between two identical runs — 126s to 142s for the same changelog — so the real
figure sits inside the noise.
Fifteen commits. The first three are the render target, the setting with its
download button and the
rel="alternate"link, and the escaping together withconfval, rubric and the text roles. The rest add one directive family each, in
the order the manuals turned them up, and close with the front matter that
tells a file which manual, version and source it came from. Four are bugfixes:
an image lost its alt text, the file text role lost its code formatting, a code
block lost the caption that says which file the snippet belongs in, and the
permalinks named no version, so every link in a 13.4 manual led to whatever is
current later.
This supersedes #1401, which added the same
rel="alternate"link pointing atthe copied reStructuredText source.
Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_019EuSAzF5qw81NzPusWy2hC